Papers with human evaluation metrics
Fine-grained Conversational Decoding via Isotropic and Proximal Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing text decoding methods are not tailoring for dialogue generation. |
| Approach: | They propose a fine-grained conversational decoding method that generates a semantic-concentrated response while maintaining informativeness and discrimination against the context. |
| Outcome: | The proposed method outperforms existing decoding strategies in the dialogue field across both automatic and human evaluation metrics. |
Read before Generate! Faithful Long Form Question Answering with Machine Reading (2022.findings-acl)
Copied to clipboard
| Challenge: | Long-form question answering (LFQA) generates a paragraph-length answer for a given question. |
| Approach: | They propose a framework that jointly models answer generation and machine reading. |
| Outcome: | The proposed model generates a more factually accurate answer from millions of documents retrieved from a large dataset. |
An Intra-Class Relation Guided Approach for Code Comment Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent work in code comment generation assumes that all information required to generate comments is encoded in the target function itself, yet in most realistic situations, it is hard to understand a function in isolation from the surrounding context. |
| Approach: | They propose a graph-based learning framework to capture various relations among functions in a class file. |
| Outcome: | The proposed method outperforms baseline models on automatic and human evaluation metrics on a Java dataset collected from real-world projects. |
A Reinforcement Learning Framework for Natural Question Generation using Bi-discriminators (C18-1)
Copied to clipboard
| Challenge: | Existing research on visual question generation is focused on training models to fit the annotated data set that makes them indifferent from other language generation tasks. |
| Approach: | They propose to use two discriminators to enhance the training of a visual question generator to ask natural questions about an image. |
| Outcome: | The proposed model outperforms state-of-the-art models in terms of automatic and human evaluation metrics. |
Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision (2022.coling-1)
Copied to clipboard
Khalil Mrini, Harpreet Singh, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter W. Chang, Emilia Farcas, Ndapa Nakashole
| Challenge: | Current medical question answering systems have difficulty processing long, detailed and informally worded questions . a growing number of approaches attempt to enhance the processing of consumer health questions - or medical question understanding . |
| Approach: | They propose a medical question understanding and answering system with knowledge grounding and semantic self-supervision that matches a user question with a trusted medical knowledge base and retrieves a fixed number of relevant sentences from the corresponding answer document. |
| Outcome: | The proposed system retrieves more relevant answers while achieving 20 times faster. |
Reinforced Multi-task Approach for Multi-hop Question Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Empirical evaluation shows our model to outperform the single-hop question generation models on both automatic evaluation metrics such as BLEU, METEOR, and ROUGE and human evaluation metrics for quality and coverage of the generated questions. |
| Approach: | They propose a question-aware reward function to maximize the utilization of supporting facts in the context. |
| Outcome: | The proposed model outperforms single-hop neural question generation models on automatic evaluation metrics and human evaluation metrics for quality and coverage of the generated questions. |
Mathematical Word Problem Generation from Commonsense Knowledge Graph and Equations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for generating mathematical word problems are lacking in educational assessment. |
| Approach: | They propose an end-to-end neural model to generate diverse mathematical word problems from commonsense knowledge graph and equations. |
| Outcome: | The proposed model outperforms the SOTA models in terms of evaluation metrics and topic relevance. |
Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention (P19-1)
Copied to clipboard
| Challenge: | Existing models for limited-domain RNNs are difficult to scale due to the complexity of the inputs. |
| Approach: | They propose to use dialog acts to build a multi-layer hierarchical graph with a disentangled self-attention network. |
| Outcome: | The proposed model improves on the baselines on automatic and human evaluation metrics. |
HeLo: Learning-Free Lookahead Decoding for Conversation Infilling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | We propose a scalable decoding strategy for conversation infilling . large pretrained language models are effective solutions to many popular natural language generation tasks such as machine translation and conversational dialogue. |
| Approach: | They propose a heuristic guided lookahead decoding strategy for conversation infilling which leverages a greedy lookalike phase before committing to any token. |
| Outcome: | The proposed strategy outperforms baselines when evaluated with automatic and human evaluation metrics, which, we argue, are appropriate for the task. |
A Survey of Pun Generation: Datasets, Evaluations and Methodologies (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pun generation aims to modify linguistic elements in text to produce humour or evoke double meanings. |
| Approach: | They propose to review pun generation datasets and methods across different stages . pun generation aims to produce humour or evoke double meanings . |
| Outcome: | This paper summarises both automated and human evaluation metrics used to assess the quality of pun generation. |
Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we generate code-tracing questions based on code snippets and descriptions. |
| Approach: | They propose to use large language models to generate code-tracing questions in introductory programming courses by using GPT4 prompts. |
| Outcome: | The proposed model generates code-tracing questions based on code snippets and descriptions. |
Type-dependent Prompt CycleQAG : Cycle Consistency for Multi-hop Question Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing research on multi-hop question generation (QG) has not been done due to its complexity. |
| Approach: | They propose a type-dependent prompt cycleQAG with a cycle consistency loss . they propose to use the question type and words related to the correct answer as prompts . |
| Outcome: | The proposed model outperforms the baseline model by 10.38% based on ROUGE score. |
DISK: Domain-constrained Instance Sketch for Math Word Problem Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for generating MWP text from equations are inflexible and require pre-defined templates. |
| Approach: | They propose a neural model which generates MWPs from equations by constructing a Quantity Cell Graph from the retrieved MWp instance and reasoning over it. |
| Outcome: | The proposed model performs impressively on educational MWP set and on human evaluation metrics. |
Don’t Go Far Off: An Empirical Study on Neural Poetry Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | despite improvements in machine translation quality, automatic poetry translation remains a challenging problem . et al., a study of automatic poetry translators shows that multilingual fine-tuning on poetic data outperforms bilingual fine-timing on non-poetic text . |
| Approach: | They propose to use poetic parallel corpora for 6 languages to study poetry translation . they find that multilingual fine-tuning on poetic data outperforms bilingual fine-uning . |
| Outcome: | The proposed model outperforms bilingual and multilingual models on poetic data . the proposed model is based on a parallel dataset of poetry translations for several languages . |
MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on fluency and factual reliability, while neglecting figurative quality. |
| Approach: | They propose a set of human evaluation metrics focused on the translation of figurative language and a parallel metaphor corpus generated by post-editing. |
| Outcome: | The proposed evaluation protocol estimates four aspects of MT: Metaphorical Equivalence, Emotion, Authenticity, and Quality. |
A Simple Yet Effective Corpus Construction Method for Chinese Sentence Compression (2022.lrec-1)
Copied to clipboard
| Challenge: | Deletion-based sentence compression has made significant progress in the english language . however, there is a lack of large-scale and high-quality parallel corpus for the Chinese language to train an efficient system. |
| Approach: | They propose to construct a Chinese corpus with 151k pairs of sentences and train extractive and generative neural compression models on the constructed corpus. |
| Outcome: | The proposed method generates high-quality compressed sentences on automatic and human evaluation metrics compared with baselines. |
Help Me Write a Story: Evaluating LLMs’ Ability to Generate Writing Feedback (2025.acl-long)
Copied to clipboard
| Challenge: | Current models provide specific and mostly accurate writing feedback, but they fail to identify the biggest writing issue in the story and to correctly decide when to offer critical vs. positive feedback. |
| Approach: | They propose a task that corrupts 1,300 stories to intentionally introduce writing issues to study model performance. |
| Outcome: | The proposed model performs well in a controlled task with human and automatic evaluation metrics. |